[io] add test bed for streaming large objects#20608
Draft
jblomer wants to merge 36 commits intoroot-project:Arlesiennefrom
Draft
[io] add test bed for streaming large objects#20608jblomer wants to merge 36 commits intoroot-project:Arlesiennefrom
jblomer wants to merge 36 commits intoroot-project:Arlesiennefrom
Conversation
silverweed
reviewed
Dec 2, 2025
|
|
||
| namespace { | ||
|
|
||
| class FileRaii { |
Contributor
There was a problem hiding this comment.
This can be pulled from ntuple_test.hxx, no?
Contributor
Author
There was a problem hiding this comment.
Yes, but perhaps just from the test support when the other PR is merged.
Contributor
Author
There was a problem hiding this comment.
I wanted to avoid including this large header here.
pcanal
force-pushed
the
io-large-buffer
branch
from
8be2fe3 to
e1f4b41
Compare
Test Results 22 files 22 suites 3d 5h 13m 43s ⏱️ For more details on these failures, see this check. Results for commit ea1e387. ♻️ This comment has been updated with latest results. |
Fixes root-project#14770 [io] add more checks in TBuffer functions as suggested by jblomer
In order to support 64 bits byte counts, which do not fit in the space reserved for them in the stream (4 bytes minus 2 control bits) and to work around the fact that the variables that holds the position and the byte count information in Streamer functions are only 32 bits (see arguments to TBufferFile::WriteVersion and TBufferFile::ReadVersion), we need to pass them indirectly when they needs 64 bits Since the streaming is inherently serial, we can leverage the sequence of calls and cache the 64 bits values in a queue. The bytecount that can not be stored in place inside the io stream will be held in a collection (fByteCounts) that need to be stored externally to the buffer.
Note to store and restore the larger than 1GB byte count use:
// Copy the content of the const reference.
auto bytecounts{b.GetByteCounts()};
...
b.SetByteCounts(std::move(bytecounts));
This fixes TBufferFile::ReadObjectAny handling of long range byte counts
Those were essentially replaced by a 'set the cursor to the end' call in commit 7d25b75. It is dubious whether these changes are needed or were an heurestic to ignore (temporarily !?) race conditions.
Due to the need for using a stack of bytecount position, we can no longer support redundant calls to CheckByteCount.
This simplify the implementation of calling `ReserveByteCount` and then `SetByteCount` and could also simplify removing the use of the ByteCountStack in some/most cases. Note: `ReserveByteCount` is protected and the existing code that does the same semantic action was only recording the position in a 32 bits integer. That code needs to be updated by either making `ReserverByteCount` public (making removal of external use of the ByteCountStack more difficult) or by introducing a RAII that hides the size of the integer.
In order to allow using the roottest CMake macro outside of the roottest sub-directory, we need to make the variable used by those macros 'cache variable' so that they can use elsewhere. This is a hack and works only after the second CMake configuration/invocation.
When using the Microsoft Visual Studio generator we have the exact same problem, that it might trigger a rebuild of ROOT so we need to apply the same solution. See 06e00a2. PS. We might want to eventually rename the resource lock.
It is need by at least Ninja and the Visual Studio generator
Uses RStreamerField as a test bed for (de-)serializing large objects with TBufferFile.
Member
|
The next step for this PR is the implementation of storage of the storage and retrieval of the large byte count array. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.